Physics based graphical diagram editor

ABSTRACT

System and method for editing a graphical diagram. A graphical diagram, such as a graphical program, is displayed on a display device. User input may be received editing the graphical diagram, thereby generating an edited graphical diagram. Placement of one or more elements in the graphical diagram may be adjusted in response to the editing based on determined forces applied to the one or more elements in the edited graphical diagram based on the said editing, resulting in an adjusted edited graphical diagram. The adjusted edited graphical diagram may be displayed on the display device, which may include displaying an animation illustrating the movement of the elements to an equilibrium state in which the forces balance and movement ceases. The editing, adjusting, and displaying may be performed sequentially and/or concurrently, as desired.

CONTINUATION AND PRIORITY DATA

This application is a continuation of U.S. application Ser. No.13/783,967, titled “Physics Based Diagram Editor”, filed Mar. 4, 2013,whose inventor was Jeffrey L. Kodosky, and which claims benefit ofpriority to U.S. Provisional Application Ser. No. 61/679,274, titled“Physics Based Graphical Program Editor”, filed Aug. 3, 2012, whoseinventor was Jeffrey L. Kodosky, which are both hereby incorporated byreference in their entirety as though fully and completely set forthherein.

FIELD OF THE INVENTION

The present invention relates to the field of graphical diagrams andgraphical editing, and more particularly to a system and method forphysics based editing of a graphical diagram, such as a graphicalprogram.

DESCRIPTION OF THE RELATED ART

Graphical programming has become a powerful tool available toprogrammers. Graphical programming environments such as the NationalInstruments LabVIEW product have become very popular. Tools such asLabVIEW have greatly increased the productivity of programmers, andincreasing numbers of programmers are using graphical programmingenvironments to develop their software applications. In particular,graphical programming tools are being used for test and measurement,data acquisition, process control, man machine interface (MMI),supervisory control and data acquisition (SCADA) applications, modeling,simulation, image processing/machine vision applications, and motioncontrol, among others.

However, given the two-dimensional (2D) nature of most graphicalprograms, creating and editing a graphical program often involvesignificant effort and time on the part of the developer to make theblock diagram of the graphical program clear and readily understandable,e.g., to make it neat and compact by aligning graphical program elements(e.g., nodes) to keep the connecting wires between them straight. Thisaspect of development is tedious when editing on a desktop computer(e.g., workstation), and is even more problematic when editing agraphical program via a touch screen, e.g., on a tablet computer, suchas an iPad™, provided by Apple, Incorporated. Similar issues applygenerally to (graphical) diagrams in general, e.g., system diagrams,configuration diagrams, or any other graphical diagrams wherein icons ornodes are connected by lines or curves.

Thus, improved techniques for editing diagrams, e.g., graphical programsare desired.

SUMMARY OF THE INVENTION

Various embodiments of a system and method for physics based editing ofa diagram, such as a graphical program, are presented below. Thetechniques disclosed herein are primarily illustrated in the figures interms of graphical programs, but are generally applicable to anygraphical diagrams wherein nodes or icons (i.e., graphical diagramelements) are interconnected with lines or curves, e.g., systemdiagrams, configuration diagrams, graphs, and so forth.

A diagram, e.g., a graphical diagram, configuration diagram, systemdiagram, graph, etc., may be displayed on a display device, e.g., of acomputer system, e.g., a pad or tablet computer, workstation, laptop,etc., as desired. The graphical diagram may be created or assembled bythe user arranging on a display a plurality of nodes or icons and theninterconnecting the nodes or icons to create the graphical diagram.

User input may be received (e.g., to an editor, e.g., a diagram editor,such as a graphical program editor) editing the graphical diagram,thereby generating an edited graphical diagram. In other words, the userinput may specify or implement an edit operation in the graphicaldiagram. Depending on the development platform, the user input may beprovided in any number of ways, including, but not limited to, any of: apointing device, such as a pointing device (e.g., mouse, stylus,trackball, etc.), a keyboard, a computer touchscreen (including, forexample, a pad or tablet computer with touch interface), a computertouchpad, and so forth, as desired.

Placement of one or more elements in the edited graphical diagram may beadjusted based on the editing, resulting in an adjusted edited graphicaldiagram. For example, one or more (simulated) forces may be applied toone or more elements of the edited graphical diagram based on theediting, or, said another say, in response to the edit operationspecified by the user input, e.g., in response to the editing by theuser. In other words, based on the editing, one or more forces may becomputed and applied to one or more elements in the edited graphicaldiagram, thereby adjusting placement of the one or more elements withinthe edited graphical diagram, resulting in an adjusted edited graphicaldiagram. Stated yet another way, the method may operate to calculatemovement(s) and/or corresponding change(s) in location (possiblyincluding size) of one or more nodes, wherein the calculated movement(s)and change(s) in location are determined by a physics based model withsimulated forces. Thus the method calculates these forces and the abovementioned movement(s)/change(s) are determined based on these calculatedforces.

Note that the determined forces are not actual physical forces appliedto actual physical objects, but rather are simulated forces applied todiagram elements with simulated physical attributes, such as mass,charge, etc. There are a number of ways such a physics based scheme maybe implemented and applied. For example, at a simple level, elements inor of the graphical diagram may repel or attract each other according tosome specified relationship. In one embodiment, nodes may repel eachother, and wires may attract each other, although more complex schemesmay be used as desired. Via judicious use (and customization) of variousphysics principles, laws or rules may be defined or specified that mayautomatically keep a diagram compact and wires aligned, and maydynamically adjust the diagram (elements) as elements are added,removed, resized, moved, etc.

The adjusted edited graphical diagram may be displayed on the displaydevice. Said another way, the result of the edit operation andadjustment or rearrangement of the diagram elements may be indicated inthe displayed graphical diagram. In some embodiments, the adjustment ormovement (or other graphically apparent modification) of the elementsmay be animated, e.g., the adjustment may be illustrated dynamically tothe user. In other words, an animation may be displayed illustrating themovement of the elements to an equilibrium state in which the forcesbalance and movement ceases.

Note that in some embodiments, at least a portion of the editing,adjusting, and displaying the adjusted edited graphical program may beperformed concurrently, e.g., in an interleaved manner. For example, inan exemplary case where the user input moves a node from a firstlocation to a second location, as soon as the node's movement begins,resulting net forces may be determined and the adjusting may also beginaccordingly, e.g., other nodes may start moving in response to themoving of the node, and may also be displayed accordingly, and so on inan iterative manner. When the node reaches the second location, theforce determination, adjustments, and display may be continued, therebyallowing the diagram elements to reach equilibrium. As noted below, insome embodiments, the force determination and displaying of the adjusteddiagram may be performed continuously even after equilibrium is reached,but since the forces are balanced (due to equilibrium being attained),no adjustments, and thus, no changes in the display, may occur, and thusit may not be apparent to the user that the physics simulation is stillexecuting.

In other embodiments, the editing, adjusting, and displaying may bediscrete events or operations. For example, the user may edit thegraphical diagram, e.g., moving a node to a new location, after whichthe forces may be calculated and applied “behind the scenes”, and onceequilibrium is reached, the adjusted diagram may be displayed. Thus,from the user's perspective, the displayed diagram may not evolvesmoothly, even though the underlying or associated model may do so,i.e., the displayed diagram may “jump” from the edited diagram image tothe final adjusted diagram image in which all motion of diagram elementshas already ceased.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained when thefollowing detailed description of the preferred embodiment is consideredin conjunction with the following drawings, in which:

FIG. 1A illustrates a computer system configured to edit and adjust agraphical diagram according to an embodiment of the present invention;

FIG. 1B illustrates a network system comprising two or more computersystems that may implement an embodiment of the present invention;

FIG. 2A illustrates an instrumentation control system according to oneembodiment of the invention;

FIG. 2B illustrates an industrial automation system according to oneembodiment of the invention;

FIG. 3A is a high level block diagram of an exemplary system which mayexecute or utilize graphical diagrams, e.g., graphical programs;

FIG. 3B illustrates an exemplary system which may perform control and/orsimulation functions utilizing graphical diagrams, e.g., graphicalprograms;

FIG. 4 is an exemplary block diagram of the computer systems of FIGS.1A, 1B, 2A and 2B and 3B;

FIG. 5 is a flowchart diagram illustrating one embodiment of a methodfor physics based editing of a graphical diagram;

FIG. 6 illustrates an exemplary block diagram of a graphical program,according to one embodiment;

FIG. 7 illustrates repulsion between nodes in a graphical diagrameditor, according to one embodiment;

FIG. 8 illustrates dependent node repulsion in a graphical diagrameditor, according to one embodiment;

FIG. 9 illustrates wire segment repulsion in a graphical diagram editor,according to one embodiment;

FIG. 10 illustrates wire joint attraction in a graphical diagram editor,according to one embodiment;

FIG. 11 illustrates boundary repulsion in a graphical diagram editor,according to one embodiment;

FIG. 12 illustrates boundary contraction in a graphical diagram editor,according to one embodiment;

FIGS. 13A and 13B illustrate movement of a graphical diagram element,and subsequent rearrangement of diagram elements, according to oneembodiment;

FIGS. 14A-14C illustrate enlargement of a graphical diagram element,including display of an affordance and terminals, connection of theelement to another element, and subsequent rearrangement of diagramelements, including compaction of the graphical diagram element,according to one embodiment; and

FIG. 15 illustrates an exemplary system diagram, according to oneembodiment.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and are herein described in detail. It should beunderstood, however, that the drawings and detailed description theretoare not intended to limit the invention to the particular formdisclosed, but on the contrary, the intention is to cover allmodifications, equivalents and alternatives falling within the spiritand scope of the present invention as defined by the appended claims.

DETAILED DESCRIPTION OF THE INVENTION Incorporation by Reference

The following references are hereby incorporated by reference in theirentirety as though fully and completely set forth herein:

U.S. application Ser. No. 13/783,967, titled “Physics Based DiagramEditor”, filed Mar. 4, 2013.

U.S. Provisional Application Ser. No. 61/679,274, titled “Physics BasedGraphical Program Editor”, filed Aug. 3, 2012.

U.S. patent application Ser. No. 12/720,966 titled “Multi-Touch Editingin a Graphical Programming Language”, filed Mar. 10, 2010.

U.S. patent application Ser. No. 12/572,455, titled “Editing a GraphicalData Flow Program in a Browser,” filed Oct. 2, 2009.

U.S. Pat. No. 7,987,445 titled “Comparing a Configuration Diagram to aReal System”, filed Jan. 10, 2006, issued on Jul. 26, 2011. U.S. Pat.No. 4,914,568 titled “Graphical System for Modeling a Process andAssociated Method,” issued on Apr. 3, 1990.

U.S. Pat. No. 5,481,741 titled “Method and Apparatus for ProvidingAttribute Nodes in a Graphical Data Flow Environment”, filed Sep. 22,1993, issued on Jan. 2, 1996.

U.S. Pat. No. 6,173,438 titled “Embedded Graphical Programming System”filed Aug. 18, 1997.

U.S. Pat. No. 6,219,628 titled “System and Method for Configuring anInstrument to Perform Measurement Functions Utilizing Conversion ofGraphical Programs into Hardware Implementations,” filed Aug. 18, 1997.

U.S. Pat. No. 7,210,117 titled “System and Method for ProgrammaticallyGenerating a Graphical Program in Response to Program Information,”filed Dec. 20, 2000.

Terms

The following is a glossary of terms used in the present application:

Memory Medium—Any of various types of memory devices or storage devices.The term “memory medium” is intended to include an installation medium,e.g., a CD-ROM, floppy disks 104, or tape device; a computer systemmemory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM,Rambus RAM, etc.; a non-volatile memory such as a Flash, magnetic media,e.g., a hard drive, or optical storage; registers, or other similartypes of memory elements, etc. The memory medium may comprise othertypes of memory as well or combinations thereof. In addition, the memorymedium may be located in a first computer in which the programs areexecuted, or may be located in a second different computer whichconnects to the first computer over a network, such as the Internet. Inthe latter instance, the second computer may provide programinstructions to the first computer for execution. The term “memorymedium” may include two or more memory mediums which may reside indifferent locations, e.g., in different computers that are connectedover a network.

Carrier Medium—a memory medium as described above, as well as a physicaltransmission medium, such as a bus, network, and/or other physicaltransmission medium that conveys signals such as electrical,electromagnetic, or digital signals.

Programmable Hardware Element—includes various hardware devicescomprising multiple programmable function blocks connected via aprogrammable interconnect. Examples include FPGAs (Field ProgrammableGate Arrays), PLDs (Programmable Logic Devices), FPOAs (FieldProgrammable Object Arrays), and CPLDs (Complex PLDs). The programmablefunction blocks may range from fine grained (combinatorial logic or lookup tables) to coarse grained (arithmetic logic units or processorcores). A programmable hardware element may also be referred to as“reconfigurable logic”.

Software Program—the term “software program” is intended to have thefull breadth of its ordinary meaning, and includes any type of programinstructions, code, script and/or data, or combinations thereof, thatmay be stored in a memory medium and executed by a processor. Exemplarysoftware programs include programs written in text-based programminglanguages, such as C, C++, PASCAL, FORTRAN, COBOL, JAVA, assemblylanguage, etc.; graphical programs (programs written in graphicalprogramming languages); assembly language programs; programs that havebeen compiled to machine language; scripts; and other types ofexecutable software. A software program may comprise two or moresoftware programs that interoperate in some manner. Note that variousembodiments described herein may be implemented by a computer orsoftware program. A software program may be stored as programinstructions on a memory medium.

Hardware Configuration Program—a program, e.g., a netlist or bit file,that can be used to program or configure a programmable hardwareelement.

Diagram—A graphical image displayed on a computer display which visuallyindicates relationships between graphical elements in the diagram.Diagrams may include configuration diagrams, system diagrams, physicaldiagrams, and/or graphical programs (among others). In some embodiments,diagrams may be executable to perform specified functionality, e.g.,measurement or industrial operations, which is represented by thediagram. Executable diagrams may include graphical programs (describedbelow) where icons connected by wires illustrate functionality of thegraphical program. Alternatively, or additionally, the diagram maycomprise a system diagram which may indicate functionality and/orconnectivity implemented by one or more devices. Various graphical userinterfaces (GUIs), e.g., front panels, may be associated with thediagram.

Program—the term “program” is intended to have the full breadth of itsordinary meaning. The term “program” includes 1) a software programwhich may be stored in a memory and is executable by a processor or 2) ahardware configuration program useable for configuring a programmablehardware element.

Graphical Program—A program comprising a plurality of interconnectednodes or icons, wherein the plurality of interconnected nodes or iconsvisually indicate functionality of the program. The interconnected nodesor icons are graphical source code for the program. Graphical functionnodes may also be referred to as blocks.

The following provides examples of various aspects of graphicalprograms. The following examples and discussion are not intended tolimit the above definition of graphical program, but rather provideexamples of what the term “graphical program” encompasses:

The nodes in a graphical program may be connected in one or more of adata flow, control flow, and/or execution flow format. The nodes mayalso be connected in a “signal flow” format, which is a subset of dataflow.

Exemplary graphical program development environments which may be usedto create graphical programs include LabVIEW®, DasyLab™, DiaDem™ andMatrixx/SystemBuild™ from National Instruments, Simulink® from theMathWorks, VEE™ from Agilent, WiT™ from Coreco, Vision Program Manager™from PPT Vision, SoftWIRE™ from Measurement Computing, Sanscript™ fromNorthwoods Software, Khoros™ from Khoral Research, SnapMaster™ from HEMData, VisSim™ from Visual Solutions, ObjectBench™ by SES (Scientific andEngineering Software), and VisiDAQ™ from Advantech, among others.

The term “graphical program” includes models or block diagrams createdin graphical modeling environments, wherein the model or block diagramcomprises interconnected blocks (i.e., nodes) or icons that visuallyindicate operation of the model or block diagram; exemplary graphicalmodeling environments include Simulink®, SystemBuild™, VisSim™,Hypersignal Block Diagram™, etc.

A graphical program may be represented in the memory of the computersystem as data structures and/or program instructions. The graphicalprogram, e.g., these data structures and/or program instructions, may becompiled or interpreted to produce machine language that accomplishesthe desired method or process as shown in the graphical program.

Input data to a graphical program may be received from any of varioussources, such as from a device, unit under test, a process beingmeasured or controlled, another computer program, a database, or from afile. Also, a user may input data to a graphical program or virtualinstrument using a graphical user interface, e.g., a front panel.

A graphical program may optionally have a GUI associated with thegraphical program. In this case, the plurality of interconnected blocksor nodes are often referred to as the block diagram portion of thegraphical program.

System Diagram—A diagram with one or more device icons and graphicalprogram code, wherein the device icons are use to specify and/orvisually indicate where different portions of graphical program code aredeployed/executed. A system diagram may indicate where (i.e., on whichsystem/device) programs or code may be executed. For example, the systemdiagram may include graphical indications showing where portions of thedisplayed graphical program code are executed. In some embodiments,various ones of the icons may represent processing elements which haveassociated programs for execution. At least one of the icons mayrepresent logical elements (e.g., executable software functions orgraphical program code). One or more of the device icons may representconfigurable elements. Thus, the system diagram may provide a systemview which allows a user to easily understand where graphical programcode is deployed among the various devices in the system.

Node—In the context of a graphical program, an element that may beincluded in a graphical program. The graphical program nodes (or simplynodes) in a graphical program may also be referred to as blocks. A nodemay have an associated icon that represents the node in the graphicalprogram, as well as underlying code and/or data that implementsfunctionality of the node. Exemplary nodes (or blocks) include functionnodes, sub-program nodes, terminal nodes, structure nodes, etc. Nodesmay be connected together in a graphical program by connection icons orwires.

Wire—a graphical element displayed in a diagram on a display thatconnects icons or nodes in the diagram. The diagram may be a graphicalprogram (where the icons correspond to software functions), a systemdiagram (where the icons may correspond to hardware devices or softwarefunctions), etc. The wire is generally used to indicate, specify, orimplement communication between the icons. Wires may represent logicaldata transfer between icons, or may represent a physical communicationmedium, such as Ethernet, USB, etc. Wires may implement and operateunder various protocols, including data flow semantics, non-data flowsemantics, etc. Some wires, e.g., buffered data transfer wires, may beconfigurable to implement or follow specified protocols or semantics.Wires may indicate communication of data, timing information, statusinformation, control information, and/or other information betweenicons. In some embodiments, wires may have different visual appearanceswhich may indicate different characteristics of the wire (e.g., type ofdata exchange semantics, data transport protocols, data transportmediums, and/or type of information passed between the icons, amongothers).

Data Flow Program—A Software Program in which the program architectureis that of a directed graph specifying the flow of data through theprogram, and thus functions execute whenever the necessary input dataare available. Data flow programs can be contrasted with proceduralprograms, which specify an execution flow of computations to beperformed. As used herein “data flow” or “data flow programs” refer to“dynamically-scheduled data flow” and/or “statically-defined data flow”.

Graphical Data Flow Program (or Graphical Data Flow Diagram)—A GraphicalProgram which is also a Data Flow Program. A Graphical Data Flow Programcomprises a plurality of interconnected nodes (blocks), wherein at leasta subset of the connections among the nodes visually indicate that dataproduced by one node is used by another node. A LabVIEW VI is oneexample of a graphical data flow program. A Simulink block diagram isanother example of a graphical data flow program.

Graphical User Interface—this term is intended to have the full breadthof its ordinary meaning. The term “Graphical User Interface” is oftenabbreviated to “GUI”. A GUI may comprise only one or more input GUIelements, only one or more output GUI elements, or both input and outputGUI elements.

The following provides examples of various aspects of GUIs. Thefollowing examples and discussion are not intended to limit the ordinarymeaning of GUI, but rather provide examples of what the term “graphicaluser interface” encompasses:

A GUI may comprise a single window having one or more GUI Elements, ormay comprise a plurality of individual GUI Elements (or individualwindows each having one or more GUI Elements), wherein the individualGUI Elements or windows may optionally be tiled together.

A GUI may be associated with a graphical program. In this instance,various mechanisms may be used to connect GUI Elements in the GUI withnodes in the graphical program. For example, when Input Controls andOutput Indicators are created in the GUI, corresponding nodes (e.g.,terminals) may be automatically created in the graphical program orblock diagram. Alternatively, the user can place terminal nodes in theblock diagram which may cause the display of corresponding GUI Elementsfront panel objects in the GUI, either at edit time or later at runtime. As another example, the GUI may comprise GUI Elements embedded inthe block diagram portion of the graphical program.

Front Panel—A Graphical User Interface that includes input controls andoutput indicators, and which enables a user to interactively control ormanipulate the input being provided to a program, and view output of theprogram, while the program is executing.

A front panel is a type of GUI. A front panel may be associated with agraphical program as described above.

In an instrumentation application, the front panel can be analogized tothe front panel of an instrument. In an industrial automationapplication the front panel can be analogized to the MMI (Man MachineInterface) of a device. The user may adjust the controls on the frontpanel to affect the input and view the output on the respectiveindicators.

Graphical User Interface Element—an element of a graphical userinterface, such as for providing input or displaying output. Exemplarygraphical user interface elements comprise input controls and outputindicators.

Input Control—a graphical user interface element for providing userinput to a program. An input control displays the value input by theuser and is capable of being manipulated at the discretion of the user.Exemplary input controls comprise dials, knobs, sliders, input textboxes, etc.

Output Indicator—a graphical user interface element for displayingoutput from a program. Exemplary output indicators include charts,graphs, gauges, output text boxes, numeric displays, etc. An outputindicator is sometimes referred to as an “output control”.

Computer System—any of various types of computing or processing systems,including a personal computer system (PC), mainframe computer system,workstation, network appliance, Internet appliance, personal digitalassistant (PDA), television system, grid computing system, or otherdevice or combinations of devices. In general, the term “computersystem” can be broadly defined to encompass any device (or combinationof devices) having at least one processor that executes instructionsfrom a memory medium.

Measurement Device—includes instruments, data acquisition devices, smartsensors, and any of various types of devices that are configured toacquire and/or store data. A measurement device may also optionally befurther configured to analyze or process the acquired or stored data.Examples of a measurement device include an instrument, such as atraditional stand-alone “box” instrument, a computer-based instrument(instrument on a card) or external instrument, a data acquisition card,a device external to a computer that operates similarly to a dataacquisition card, a smart sensor, one or more DAQ or measurement cardsor modules in a chassis, an image acquisition device, such as an imageacquisition (or machine vision) card (also called a video capture board)or smart camera, a motion control device, a robot having machine vision,and other similar types of devices. Exemplary “stand-alone” instrumentsinclude oscilloscopes, multimeters, signal analyzers, arbitrary waveformgenerators, spectroscopes, and similar measurement, test, or automationinstruments.

A measurement device may be further configured to perform controlfunctions, e.g., in response to analysis of the acquired or stored data.For example, the measurement device may send a control signal to anexternal system, such as a motion control system or to a sensor, inresponse to particular data. A measurement device may also be configuredto perform automation functions, i.e., may receive and analyze data, andissue automation control signals in response.

Automatically—refers to an action or operation performed by a computersystem (e.g., software executed by the computer system) or device (e.g.,circuitry, programmable hardware elements, ASICs, etc.), without userinput directly specifying or performing the action or operation. Thusthe term “automatically” is in contrast to an operation being manuallyperformed or specified by the user, where the user provides input todirectly perform the operation. An automatic procedure may be initiatedby input provided by the user, but the subsequent actions that areperformed “automatically” are not specified by the user, i.e., are notperformed “manually”, where the user specifies each action to perform.For example, a user filling out an electronic form by selecting eachfield and providing input specifying information (e.g., by typinginformation, selecting check boxes, radio selections, etc.) is fillingout the form manually, even though the computer system must update theform in response to the user actions. The form may be automaticallyfilled out by the computer system where the computer system (e.g.,software executing on the computer system) analyzes the fields of theform and fills in the form without any user input specifying the answersto the fields. As indicated above, the user may invoke the automaticfilling of the form, but is not involved in the actual filling of theform (e.g., the user is not manually specifying answers to fields butrather they are being automatically completed). The presentspecification provides various examples of operations beingautomatically performed in response to actions the user has taken.

FIG. 1A—Computer System

FIG. 1A illustrates a computer system 82 configured to implementembodiments of the invention. One embodiment of a method for physicsbased editing of a graphical diagram is described below.

As shown in FIG. 1A, the computer system 82 may include a display deviceconfigured to display the graphical diagram as the graphical program iscreated and/or edited, and in embodiments where the diagram is agraphical program, possibly executed. For example, the display devicemay display a graphical user interface (GUI) of a graphical diagrameditor or development environment, e.g., a graphical programmingdevelopment environment application used to create, edit, and/or executegraphical programs, such as the LabVIEW™ graphical program developmentenvironment provided by National Instruments Corporation. The graphicalprogram development environment may be configured to utilize or supportphysics based operations for developing graphical programs. The displaydevice may also be configured to display a graphical user interface orfront panel of the graphical program during execution of the graphicalprogram. The graphical user interface(s) may comprise any type ofgraphical user interface, e.g., depending on the computing platform.

The computer system 82 may include at least one memory medium on whichone or more computer programs or software components according to oneembodiment of the present invention may be stored. For example, thememory medium may store one or more programs, e.g., graphical programs,which are executable to perform the methods described herein.Additionally, the memory medium may store a diagram editor for creatingand editing graphical diagrams, e.g., graphical programming developmentenvironment application used to create and/or execute graphicalprograms. The memory medium may also store operating system software, aswell as other software for operation of the computer system. Variousembodiments further include receiving or storing instructions and/ordata implemented in accordance with the foregoing description upon acarrier medium.

FIG. 1B—Computer Network

FIG. 1B illustrates a system including a first computer system 82 thatis coupled to a second computer system 90. The computer system 82 may becoupled via a network 84 (or a computer bus) to the second computersystem 90. The computer systems 82 and 90 may each be any of varioustypes, as desired. The network 84 can also be any of various types,including a LAN (local area network), WAN (wide area network), theInternet, or an Intranet, among others. In some embodiments, thegraphical diagram editor or development environment, e.g., graphicalprogram development environment, may be configured to operate in adistributed manner. For example, the development environment may behosted or executed on the second computer system 90, while the GUI forthe development environment may be displayed on the computer system 82,and the user may create and edit a graphical diagram or program over thenetwork. In another embodiment, the development environment may beimplemented as a browser-based application. For example, the user uses abrowser program executing on the computer system 82 to access anddownload the development environment and/or graphical diagram or programfrom the second computer system 90 to create and/or edit the graphicaldiagram or program, where the development environment may execute withinthe user's browser. Further details regarding such browser-based editingof graphical programs are provided in U.S. patent application Ser. No.12/572,455, titled “Editing a Graphical Data Flow Program in a Browser,”filed Oct. 2, 2009, which was incorporated by reference above.

In some embodiments where the diagram is a graphical program, thecomputer systems 82 and 90 may execute a graphical program in adistributed fashion. For example, computer 82 may execute a firstportion of the block diagram of a graphical program and computer system90 may execute a second portion of the block diagram of the graphicalprogram. As another example, computer 82 may display the graphical userinterface of a graphical program and computer system 90 may execute theblock diagram of the graphical program. In one embodiment, the graphicaluser interface of the graphical program may be displayed on a displaydevice of the computer system 82, and the block diagram may execute on adevice coupled to the computer system 82. The device may include aprogrammable hardware element and/or may include a processor and memorymedium which may execute a real time operating system. In oneembodiment, the graphical program may be downloaded and executed on thedevice. For example, an application development environment with whichthe graphical program is associated may provide support for downloadinga graphical program for execution on the device in a real time system.

Exemplary Systems

Embodiments of the present invention may be involved with performingtest and/or measurement functions; controlling and/or modelinginstrumentation or industrial automation hardware; modeling andsimulation functions, e.g., modeling or simulating a device or productbeing developed or tested, etc. Exemplary test applications where thegraphical diagram or program may be used include hardware-in-the-looptesting and rapid control prototyping, among others.

However, it is noted that embodiments of the present invention can beused for a plethora of applications and is not limited to the aboveapplications. In other words, applications discussed in the presentdescription are exemplary only, and embodiments of the present inventionmay be used in any of various types of systems. Thus, embodiments of thesystem and method of the present invention is configured to be used inany of various types of applications, including the control of othertypes of devices such as multimedia devices, video devices, audiodevices, telephony devices, Internet devices, etc., as well as generalpurpose software applications such as word processing, spreadsheets,network control, network monitoring, financial applications, games, etc.

FIG. 2A illustrates an exemplary instrumentation control system 100which may implement embodiments of the invention. The system 100comprises a host computer 82 which couples to one or more instruments.The host computer 82 may comprise a CPU, a display screen, memory, andone or more input devices such as a mouse or keyboard as shown. Thecomputer 82 may operate with the one or more instruments to analyze,measure, or control a unit under test (UUT) or process 150.

The one or more instruments may include a GPIB instrument 112 andassociated GPIB interface card 122, a data acquisition board 114inserted into or otherwise coupled with chassis 124 with associatedsignal conditioning circuitry 126, a VXI instrument 116, a PXIinstrument 118, a video device or camera 132 and associated imageacquisition (or machine vision) card 134, a motion control device 136and associated motion control interface card 138, and/or one or morecomputer based instrument cards 142, among other types of devices. Thecomputer system may couple to and operate with one or more of theseinstruments. The instruments may be coupled to the unit under test (UUT)or process 150, or may be coupled to receive field signals, typicallygenerated by transducers. The system 100 may be used in a dataacquisition and control application, in a test and measurementapplication, an image processing or machine vision application, aprocess control application, a man-machine interface application, asimulation application, or a hardware-in-the-loop validationapplication, among others.

FIG. 2B illustrates an exemplary industrial automation system 160 whichmay implement embodiments of the invention. The industrial automationsystem 160 is similar to the instrumentation or test and measurementsystem 100 shown in FIG. 2A. Elements which are similar or identical toelements in FIG. 2A have the same reference numerals for convenience.The system 160 may comprise a computer 82 which couples to one or moredevices or instruments. The computer 82 may comprise a CPU, a displayscreen, memory, and one or more input devices such as a mouse orkeyboard as shown. The computer 82 may operate with the one or moredevices to perform an automation function with respect to a process ordevice 150, such as MMI (Man Machine Interface), SCADA (SupervisoryControl and Data Acquisition), portable or distributed data acquisition,process control, advanced analysis, or other control, among others.

The one or more devices may include a data acquisition board 114inserted into or otherwise coupled with chassis 124 with associatedsignal conditioning circuitry 126, a PXI instrument 118, a video device132 and associated image acquisition card 134, a motion control device136 and associated motion control interface card 138, a fieldbus device170 and associated fieldbus interface card 172, a PLC (ProgrammableLogic Controller) 176, a serial instrument 182 and associated serialinterface card 184, or a distributed data acquisition system, such asthe FieldPoint system available from National Instruments, among othertypes of devices.

FIG. 3A is a high level block diagram of an exemplary system which mayexecute or utilize graphical diagrams, e.g., graphical programs. FIG. 3Aillustrates a general high-level block diagram of a generic controland/or simulation system which comprises a controller 92 and a plant 94.The controller 92 represents a control system/algorithm the user may betrying to develop. The plant 94 represents the system the user may betrying to control. For example, if the user is designing an ECU for acar, the controller 92 is the ECU and the plant 94 is the car's engine(and possibly other components such as transmission, brakes, and so on.)As shown, a user may create a graphical program that specifies orimplements the functionality of one or both of the controller 92 and theplant 94. For example, a control engineer may use a modeling andsimulation tool to create a model (graphical program) of the plant 94and/or to create the algorithm (graphical program) for the controller92.

FIG. 3B illustrates an exemplary system which may perform control and/orsimulation functions. As shown, the controller 92 may be implemented bya computer system 82 or other device (e.g., including a processor andmemory medium and/or including a programmable hardware element) thatexecutes or implements a graphical program. In a similar manner, theplant 94 may be implemented by a computer system or other device 144(e.g., including a processor and memory medium and/or including aprogrammable hardware element) that executes or implements a graphicalprogram, or may be implemented in or as a real physical system, e.g., acar engine.

In one embodiment of the invention, one or more graphical programs maybe created which are used in performing rapid control prototyping. RapidControl Prototyping (RCP) generally refers to the process by which auser develops a control algorithm and quickly executes that algorithm ona target controller connected to a real system. The user may develop thecontrol algorithm using a graphical program, and the graphical programmay execute on the controller 92, e.g., on a computer system or otherdevice. The computer system 82 may be a platform that supports real timeexecution, e.g., a device including a processor that executes a realtime operating system (RTOS), or a device including a programmablehardware element.

In one embodiment of the invention, one or more graphical programs maybe created which are used in performing Hardware in the Loop (HIL)simulation. Hardware in the Loop (HIL) refers to the execution of theplant model 94 in real time to test operation of a real controller 92.For example, once the controller 92 has been designed, it may beexpensive and complicated to actually test the controller 92 thoroughlyin a real plant, e.g., a real car. Thus, the plant model (implemented bya graphical program) is executed in real time to make the realcontroller 92 “believe” or operate as if it is connected to a realplant, e.g., a real engine.

In the embodiments of FIGS. 2A, 2B, and 3B above, one or more of thevarious devices may couple to each other over a network, such as theInternet. In one embodiment, the user operates to select a target devicefrom a plurality of possible target devices for programming orconfiguration using a graphical program. Thus the user may create agraphical program on a computer and use (execute) the graphical programon that computer or deploy the graphical program to a target device (forremote execution on the target device) that is remotely located from thecomputer and coupled to the computer through a network.

Graphical software programs which perform data acquisition, analysisand/or presentation, e.g., for measurement, instrumentation control,industrial automation, modeling, or simulation, such as in theapplications shown in FIGS. 2A and 2B, may be referred to as virtualinstruments.

Note that some graphical diagrams may include a graphical program. Forexample, in some embodiments, a (graphical) system diagram may includehardware device icons and software or program icons that implement orrepresent graphical programs, e.g., the system diagram may specify whichprograms, e.g., graphical programs, are deployed, or are to be deployed,to which hardware devices. Moreover, in some embodiments, such systemdiagrams may be executable to deploy the programs to the specifieddevices, and possibly even invoke execution of the programs on thosedevices, e.g., may invoke execution of the distributed system specifiedby the system diagram.

FIG. 4—Computer System Block Diagram

FIG. 4 is a block diagram representing one embodiment of the computersystem 82 and/or 90 illustrated in FIGS. 1A and 1B, or computer system82 shown in FIG. 2A or 2B. It is noted that any type of computer systemconfiguration or architecture can be used as desired, and FIG. 4illustrates a representative PC embodiment. It is also noted that thecomputer system may be a general purpose computer system, a computerimplemented on a card installed in a chassis, or other types ofembodiments. Elements of a computer not necessary to understand thepresent description have been omitted for simplicity.

The computer may include at least one central processing unit or CPU(processor) 160 which is coupled to a processor or host bus 162. The CPU160 may be any of various types, including an x86 processor, e.g., aPentium class, a PowerPC processor, a CPU from the SPARC family of RISCprocessors, as well as others. A memory medium, typically comprising RAMand referred to as main memory, 166 is coupled to the host bus 162 bymeans of memory controller 164. The main memory 166 may store thegraphical diagram (e.g., graphical program) development environmentconfigured to utilize or support physics based edit operations, andgraphical diagrams (e.g., graphical programs) developed thereby. Themain memory may also store operating system software, as well as othersoftware for operation of the computer system.

The host bus 162 may be coupled to an expansion or input/output bus 170by means of a bus controller 168 or bus bridge logic. The expansion bus170 may be the PCI (Peripheral Component Interconnect) expansion bus,although other bus types can be used. The expansion bus 170 includesslots for various devices such as described above. The computer 82further comprises a video display subsystem 180 and hard drive 182coupled to the expansion bus 170. The computer 82 may also comprise aGPIB card 122 coupled to a GPIB bus 112, and/or an MXI device 186coupled to a VXI chassis 116.

As shown, a device 190 may also be connected to the computer. The device190 may include a processor and memory which may execute a real timeoperating system. The device 190 may also or instead comprise aprogrammable hardware element. The computer system may be configured todeploy a program, e.g., a graphical program, implementing thefunctionality disclosed herein, to the device 190. The deployed programmay take the form of a circuit design/implementation, e.g., a configuredprogrammable hardware element or ASIC. In some embodiments, the deployedprogram may take the form of graphical program instructions or datastructures that directly represent the graphical program. Alternatively,the deployed graphical program may take the form of text code (e.g., Ccode) generated from the graphical program. As another example, thedeployed graphical program may take the form of compiled code generatedfrom either the graphical program or from text code that in turn wasgenerated from the graphical program. However, note that deployment tohardware (e.g., programmable hardware element or ASIC) provides benefitsthat may not be obtained from a CPU based implementation.

FIG. 5—Flowchart of a Method for Editing a Graphical Diagram

FIG. 5 illustrates a method for editing a graphical diagram (or simply“diagram”), e.g., a graphical program, system diagram, configurationdiagram, or any other type of diagram or graph desired, using physicsbased edit operations. The method shown in FIG. 5 may be used inconjunction with any of the computer systems or devices shown in theabove Figures, among other devices. In various embodiments, some of themethod elements shown may be performed concurrently, in a differentorder than shown, or may be omitted. Additional method elements may alsobe performed as desired. As shown, this method may operate as follows.

First, in 502 a graphical diagram, e.g., a graphical program, may bedisplayed on a display device, e.g., of the computer system 82 (or on adifferent computer system). The graphical diagram may be created orassembled by the user arranging on a display a plurality of nodes oricons and then interconnecting the nodes to create the graphicalprogram. In response to the user assembling the graphical diagram, datastructures may be created and stored which represent the graphicaldiagram. In embodiments where the graphical diagram is a graphicalprogram, the nodes may be interconnected in one or more of a data flow,control flow, or execution flow format. The graphical program may thuscomprise a plurality of interconnected nodes or icons which visuallyindicates the functionality of the program. As noted above, thegraphical program may comprise a block diagram and may also include auser interface portion or front panel portion. Where the graphicalprogram includes a user interface portion, the user may optionallyassemble the user interface on the display. As one example, the user mayuse the LabVIEW graphical programming development environment to createthe graphical program. The graphical programming development environmentmay be configured to support physics based editing operations, as willbe described in more detail below.

In an alternate embodiment, the graphical program may be created in 502by the user creating or specifying a prototype, followed by automatic orprogrammatic creation of the graphical program from the prototype. Thisfunctionality is described in U.S. patent application Ser. No.09/587,682 titled “System and Method for Automatically Generating aGraphical Program to Perform an Image Processing Algorithm”, which ishereby incorporated by reference in its entirety as though fully andcompletely set forth herein. The graphical program may be created inother manners, either by the user or programmatically, as desired. Thegraphical program may implement a measurement function that is desiredto be performed by the instrument.

FIG. 6 illustrates a simple exemplary graphical diagram, specifically, agraphical program 600, according to one embodiment. As may be seen, thisexample graphical program includes various interconnected graphicalprogram nodes, including a “histogram node” 602, an “add node” 604, a“multiply node” 606, a “subtract node” 608, and an “absolute value node”610. Note that the nodes shown are exemplary only, and that any othergraphical program nodes or structures may be used as desired, e.g., loopnodes or structures that include a frame within which other nodes may beplaced, such as graphical WHILE loops, graphical FOR loops, and soforth. Other examples of nodes or structures with frames include agraphical case statement, a graphical sequence structure, and agraphical conditional structure, among others. Various exemplarygraphical program elements, such as those shown in the graphical programof FIG. 6, and variants thereof, will be used to illustrate variousexemplary physics based edit operations, described below with referenceto FIGS. 7-12. However, it should be noted that the same or similartechniques may be applied to other types of graphical diagrams.

In 504, user input may be received (e.g., to an editor, such as agraphical program editor) editing the graphical diagram, therebygenerating an edited graphical diagram. In other words, the user inputmay specify or implement an edit operation in the graphical diagram.Depending on editor or the development platform, the user input may beprovided in any number of ways, including, but not limited to, any of: apointing device, such as a pointing device (e.g., mouse, stylus,trackball, etc.), a keyboard, a computer touchscreen (including, forexample, a pad or tablet computer with touch interface), a computertouchpad, and so forth, as desired.

In 506, placement of one or more elements in the edited graphicaldiagram may be adjusted ba, resulting in an adjusted edited graphicaldiagram. For example, one or more (simulated) forces may be applied toone or more elements of the edited graphical diagram in response to theuser input, or, said another say, in response to the edit operationspecified by the user input, e.g., in response to the edit. In otherwords, based on the user input (or edit operation), one or more forcesmay be computed and applied to one or more elements in the editedgraphical diagram, thereby adjusting placement (e.g., respectivepositions, possibly including boundary positions) of the one or moreelements within or of the edited graphical diagram (or other geometricattributes of diagram elements or the diagram in general), resulting inan adjusted edited graphical diagram. Stated yet another way, the methodmay operate to calculate movement(s) and/or corresponding change(s) inlocation (possibly including size or shape) of one or more nodes orother elements, or the diagram at large, wherein the calculatedmovement(s) and change(s) in location are determined by a physics basedmodel with simulated forces. Thus the method calculates these forces andthe above mentioned movement(s)/change(s) are determined based on thesecalculated forces. In other words, the method may operate to calculatechanges in one or more of the nodes in the diagram, e.g., movement(s)and/or corresponding change(s) in location (possibly including size) ofone or more nodes, wherein the calculated movement(s) and change(s) inlocation are determined by a physics based model with simulated forces.Thus the method calculates these forces and the above mentionedmovement(s)/change(s) are determined based on these calculated forces.

Note that the determined forces are not actual physical forces appliedto actual physical objects, but rather are simulated forces applied todiagram elements with simulated physical attributes, such as mass,charge, etc. There are a number of ways such a physics based scheme maybe implemented and applied. For example, at a simple level, elements inor of the graphical diagram may repel or attract each other according tosome specified relationship. In one embodiment, nodes may repel eachother, and wires (or more generally, edges or curves) may attract eachother, although more complex schemes may be used as desired. Viajudicious use (and customization) of various physics principles, laws orrules may be defined or specified that may automatically keep a diagramcompact and wires aligned, and may dynamically adjust the diagram(elements) as elements or added, removed, or moved.

In 508, the adjusted edited graphical diagram may be displayed on thedisplay device. Said another way, the result of the editing, includingany effects of the forces on the elements, may be indicated in thedisplayed adjusted edited graphical diagram.

In some embodiments, applying forces to the one or more elements mayinclude determining one or more forces for each of the one or moreelements based on their positions, summing the forces on each of the oneor more elements, thereby determining resultant forces, moving the oneor more elements based on the resultant forces, and repeating thedetermining, summing, and moving one or more times in an iterativemanner until an equilibrium condition obtains, i.e., until forcesbalance and the elements stop moving. In one embodiment, displaying theadjusted edited graphical diagram may include displaying an animation(or “movie”) that illustrates the movement (which may include resizing,reshaping, or otherwise changing graphical aspects or geometry) of thediagram elements (or the diagram in general, e.g., diagram boundaries,shape, and so forth), as discussed below in detail.

FIGS. 7-12—Exemplary Physics Based Edit Operations

FIGS. 7-12 illustrate various exemplary physics based edit operations ona graphical program, although it should be noted that the operationsshown are meant to be illustrative only, and are not intended to limitthe physics based edit operations to any particular set, or anyparticular type of graphical diagram. Note that in these examples, andin the example figures described below, forces are indicated by arrows,each representing a force applied to a graphical program (or moregenerally, graphical diagram) element (including program boundaries).

In one embodiment, due to unwanted long-range effects of simple Coulombtype forces (r² forces), short range forces may be specified to operatebetween elements. Such short range forces may operate in a mannersimilar to a “foam” around the elements, e.g., nodes, where, forexample, when two nodes are too close, it is “compressed”, and forcesthe nodes apart, but once the nodes are (far enough) apart the forcegoes to zero. Such an exemplary force is illustrated in FIG. 7, where,as may be seen, “expanded bounds” (the above “foam padding”) that extendbeyond the normal bounds of the nodes are defined. In response tooverlap of these specified expanded bounds, repulsive forcesproportional to the overlap area may be determined and applied to thenodes, as indicated by the arrows labeled “repulsive forces”. Thus, ifthe user “drops” a node onto an already populated block diagram, thevarious elements in the diagram may automatically rearrange themselvesto make room for the new node, and/or the new node may automaticallymove to minimize congestion in the diagram.

Note that in some embodiments, the expanded bounds, overlap area, andrepulsive force vectors (arrows shown), may not actually be shown orillustrated in the graphical diagram; however, in other embodiments, theexpanded bounds, the overlap area, and/or the resulting forces, (and/orany other aspects of the technique) may be displayed in the graphicaldiagram, e.g., to indicate to the user what is likely to result from theedit operation. Such display of these aspects may be temporary, e.g.,may disappear upon completion of the edit operation, once the forceshave been applied, etc., as desired.

As another example, in one exemplary embodiment, a short range force maybe defined or specified that moves nodes to the right of nodes theydepend on, i.e., to the right of nodes from which they receive input,i.e., “source nodes” (and/or moves the source node to the left of thedependent node). An exemplary illustration of such a force is presentedin FIG. 8. As shown, in this example, when a node (labeled “dependentnode”) is wired to receive input from another node (a “source node”,labeled simply “node”), a distance, referred to herein as a “retrogradedistance”, may be determined between a specified source reference lineand a destination reference line. These reference lines may indicate ordemarcate an expanded boundary of the nodes in a similar manner as the“foam pads” of FIG. 7, but may be limited to the sides of nodes thathave I/O connections, in this case, the left and right sides, as shown.As may be seen, the retrograde distance is simply the projected overlapbetween these expanded “connection” boundaries. The term “projected”here refers to the fact that the areas only overlap in the 1D(horizontal) projection. As also indicated in FIG. 8, a repulsive forceproportional to the determined retrograde distance may be determined andapplied to the nodes, thereby moving the dependent node to the right ofthe (source) node (and in this case, also moving the source node to theleft).

Similar techniques may be applied to wires, which connect nodes in agraphical program. For example, in one embodiment, wire segments mayrepel nodes as well as each other, with the same or similar short rangeforce. FIG. 9 illustrates wire segment repulsion with respect to a node,according to one embodiment. As FIG. 9 shows, a wire segment may haveassociated expanded segment bounds that extend beyond the wire segmentbounds. If the expanded segment bounds (or a specified portion thereof,e.g., the vertical segment shown) overlaps the nodes expanded boundary,as indicated by the overlap area shown, a repulsive force proportionalto the overlap area may be determined and applied to the node and/or thewire segment (in the illustrated embodiment, the force is applied toboth), thereby moving them apart.

As another exemplary case, wire joints may have an associated attractiveforce that operates to minimize the distance between parallel segmentsof a wire. FIG. 10 illustrates an exemplary embodiment where joint 1 andjoint 2 are adjacent joints of a vertical segment of a wire, e.g., wherethe wire may connect two nodes, not shown. Note that in general, wiresegments in a graphical program are constrained to be either vertical orhorizontal, as shown. In this example, a distance is determined betweenjoint 1 and joint 2, and an attractive force proportional to thisdistance determined and applied to the joints, and thus, theirrespective horizontal segments, thereby moving the joints/segmentscloser to each other. Note that in this example, the attractive forcemay generally increase as the distance between the joints/segmentsincreases. While the above techniques directed to connecting elementsare described with respect to graphical program wires, they are alsoapplicable to any other types of connecting elements in graphicaldiagrams, e.g., graph edges, lines, or curves connecting graphicaldiagram elements.

It should be noted that the particular force dependencies used, e.g.,proportional (and/or inversely proportional) to distance, are meant tobe exemplary only, and that any types of forces and dependencies may beused as desired. Note, for example, the inset force vs. distance plot inFIG. 10, which shows that the proportional (to distance) relationshipdescribed above may only apply at mid-range distances. As indicated, inthis exemplary case, the attractive force ceases to be linearlydependent on distance at close range and at far range. Morespecifically, as distance increases, eventually the magnitude of theresulting force levels off, as indicated by the “limit” line shown.Similarly, as the distance decreases (from mid-range), the forcemagnitude increases sharply, then drops off even faster, resulting in aforce profile that causes the segments to “grab” one another at someoptimum distance. Thus, in some embodiments, the force law or profilemay include a non-linear component at small distances. As result of thistechnique applied to wires may automatically straighten small wire“jags”. Of course, any other force profiles may be defined and used asdesired, with corresponding, possibly complex, behaviors of effectedelements.

In a further example, nodes and boundaries (e.g., of structure nodesand/or of the graphical program (or more generally, diagram) itself) mayhave associated force relationships. For example, some structure nodesinclude frames or borders that can contain other nodes, e.g., loops,sequence structures, and so forth, and so the frame has an innerboundary. In some embodiments, nodes and wire segments may repel theboundary of the enclosing structure (node) with a short range forcesimilar to those described above.

FIG. 11 illustrates an exemplary embodiment where the inside boundary ofa frame of a structure node has an associated inset boundary (or innerexpanded boundary, e.g., “foam padding” per the above-describedmetaphor). Similar to the example of FIG. 7, if a node's expandedboundary overlaps with the inset boundary of the frame, repulsive forcesproportional to the resulting overlap area (so labeled) may bedetermined and applied to the node and/or the frame. Note that in theembodiment shown, it is the left frame edge/boundary whose insetboundary overlaps, and so the resulting force applied thereto ishorizontal (and to the left). In some embodiments, such a force may movethe entire frame/structure node. However, in other embodiments, only the“overlapped” frame segment may be moved, and so the frame may bestretched in the direction of the applied force, automatically enlargingin that direction to accommodate the node. As also shown, the nodeexperiences a corresponding force to the right, and may thus moveaccordingly.

In some embodiments, such structure node frames may have associatedcontractive (or attractive) forces that may operate to “shrink wrap”contained nodes, thereby minimizing the footprint of the structure node(with contained elements). FIG. 12 illustrates an exemplary embodimentin which the inside width of the frame, labeled “width” in the figure,may invoke such contractive forces, as shown. More specifically, in thisexample, if the width exceeds a specified minimum width (so labeled),contractive forces proportional to the arctangent (atan) of thedifference between the width and the minimum width may be determined andapplied to the frame edges (in this particular case, just the left andright edges, as shown).

As illustrated by the force profile inset shown in FIG. 12, thisparticular force law specifies that the magnitude of the force goes tozero as the width approaches the minimum width, grows as the widthexceeds the minimum width by greater amounts (up to a limit, asindicated), and turns negative (thereby becomingrepulsive/expansive/dispersive) as the width becomes less than theminimum width. Thus, the frame boundary will tend to shrink if the widthexceeds the minimum width, and will tend to expand if the width becomessmaller than the minimum width (both within bounds, of course, asindicated in the force profile, and as is the case with the arctangentfunction).

In another embodiment, the contractive force may be constant, or almostconstant, which may prevent crushing of nested structures, e.g.,structure nodes within structure nodes.

It should be noted that the above force schemes, laws, profiles,distances, boundaries, and element behaviors, are meant to be exemplaryonly, and that any types of force schemes, laws, profiles, distances,boundaries, and element behaviors may be used as desired, and in anycombinations.

Note further that an important benefit of using such forces is that theygenerally operate together in a smooth graceful manner. For example, twoor more of the forces and corresponding effects may operate at the sametime, possibly at odds with one another. In such cases, all forces maybe computed and summed per element (within appropriate constraints,e.g., constrained movement, force directions, etc., for specificelements), and the resultant forces may then move the elementsaccordingly. Thus, for example, in a case where a frame boundary of astructure node has a contractive force due to its width exceeding theminimum width, but contains nodes whose inter-node repulsive forcesprevent them from clustering within the minimum width, and where theframe and the nodes are repelled by each other, an equilibrium may bereached that results in a compromise between the specified behaviors.

In some embodiments, the forces may be applied over time. In otherwords, rather than simply determining the forces and moving the elementsin a single step, the method may utilize dynamics to reach equilibriumamong the elements, i.e., applying forces to and moving the elementsuntil an equilibrium condition obtains and thus, movement ceases.Elements may be assigned masses to modulate or control the appliedforces. Moreover, in order to prevent oscillations, which can be commonwith complex force based systems, damping may be employed, e.g., afriction-like effect that operates to slowly lower the kinetic energy(which may be considered “temperature”) of the system to zero. In someembodiments, such damping may be tuned to optimize convergence toequilibrium, referred to as critical damping. Exemplary dynamicalequations or relationships contemplated include:Δv=Δt*force/mass  (1)v=damping*(v+Δv)  (2)position′=position+Δt*v  (3)

where v is velocity, and t is time. However, it should be noted that anyother dynamical relationships may be used as desired, as the techniquesdisclosed herein are not limited to physically real dynamics, e.g.,“real physics”. Rather, any type of force laws and relationships may bedefined and used as desired.

Thus, in some embodiments, the editor may create and execute a dynamicalmodel or simulation of the block diagram (graphical program or othergraphical diagram), simulating physical interactions between theelements. The model may specify and implement physics based attributesand behaviors of the graphical program or graphical program elements viaa plurality of physics based parameters, as discussed below in moredetail. The model may run continuously in a loop, calculating forces onall elements, e.g., nodes and wire joints, updating velocities andupdating positions. In one embodiment, the mass of nodes may be greaterthan that of wire joints so joints will move faster and adjust morequickly.

Accordingly, once the user has edited the diagram, or has begun the editoperation, the editor may display an animation or “movie” of the diagramas the elements automatically adjust, e.g., as the forces are appliedand elements are moved accordingly. In other words, displaying theadjusted edited graphical program on the display device may includedisplaying an animation that illustrates the moving (which may includeresizing, reshaping, etc.) of the diagram elements, possibly includingdiagram boundaries, shape, etc. In one embodiment, the frame rate of themovie may be equal to the update cycle of the model, although in otherembodiments, the frame rate of the movie may differ from the update (orcycle) rate of the model. For example, the model may cycle much fasterthan 30 frames per second (fps), but the movie may have a frame rate ator close to 30 fps. In one exemplary embodiment, the movie frame ratemay be variable, depending on the rate of change in the positions (orplacements) of the elements, e.g., the velocities of the elements,where, for example, the frame rate may be faster when the elements movequickly, and may be slower when they move more slowly.

Note that since nodes may generally repel each other, they may notspontaneously move past each other, with the result that the diagram canbe stable in multiple different arrangements. Moreover, due to theautomatic equilibrium seeking behavior of the collective system, theuser may adjust the gross layout, and the physics may “tidy” up thediagram as well as it can. Similarly, a user may expand or contractitems in the block diagram (i.e., converting an item between a singleicon representation (e.g., a subprogram node) and a multi-iconpresentation of the item's constituent elements, e.g., the subprogram),and the elements may automatically adjust accordingly. This feature maybe particularly useful with small display screens, e.g., tabletcomputers or even smart phones. Further embodiments regarding therelationship between the model and the animation of the graphicaldiagram are described below.

In one exemplary use case, when a node is dragged into or out of astructure node (frame or bounding structure/element), the node repelsthe boundary or border, but because the border has inertia (due to itsmass), moving the node quickly can move the node past the border. Onceon the other side the node repels it in the opposite direction, andthus, the node makes a little bounce when it crosses the border, whichmay have a nice physical feel to it (to the user). In some embodiments,one or more dynamical effects, such as this “bounce”, may triggercorresponding sound effects, e.g., a “pop” sound, which may increase theenjoyment and thus the engagement of the user.

In another exemplary use case, such “live dragging” of nodes may beleveraged to implement wiring (between the nodes). For example, the usermay tap a node to make terminals easier to hit, get a “plug” (e.g., aconnector) and drag it to a terminal on another node. Since the plug isjust another node, dragging it in/out of a structure to perform wiringmay not require extra functionality to be designed into the system. Thesame gesture may be used to pull a plug off a node and wire it tosomething else, or to just leave it and wire it later. Having a plugfloating in the diagram may be preferable to a “broken” wire.

Other synergies may also arise due to the complex nature of physicalinteractions among multiple elements, thereby further leveraging thetechniques disclosed herein to improve and enhance the graphical editingexperience.

As noted above, in some embodiments, an indication of the (one or more)forces or their effects may be displayed in the graphical program beforeor as the adjustment is performed. In some embodiments, such indicationsmay be displayed as the graphical diagram is edited. Moreover, in someembodiments, key distances or auxiliary aspects of the above techniquesmay be displayed, e.g., expanded boundaries, minimum widths, etc. Insome embodiments, the user may be able to invoke display of the relevantforce laws/profiles, and may further be able to edit or configure any ofthe aspects or attributes described herein. As noted above, any of thetechniques disclosed herein may be similarly applicable to othergraphical diagrams, e.g., system diagrams, physical diagrams,configuration diagrams, etc., where system components, e.g., hardwareand/or software components, are linked together in particular ways.

FIGS. 13A and 13B—Exemplary Embodiment

The following presents an exemplary illustrative example of the abovetechniques wherein the exemplary block diagram (or graphical program) ofFIG. 6 is edited, and exemplary results of applying forces are shown.More specifically, FIGS. 13A and 13B illustrate movement of a graphicalprogram element, and subsequent rearrangement of program elements,according to one embodiment.

As shown, FIG. 13A illustrates user input specifying moving node 610 tobe between nodes 606 and 608 in diagram or graphical program 1301. Notethat in this particular case, the movement is specified via a usergesture in which the user taps or otherwise selects the node 610 with afinger (represented by a dashed circle overlaid on node 610) and moves,e.g., drags, the node to the new location (represented by the seconddashed circle disposed between nodes 606 and 608). However, anytechniques specifying edits may be used as desired, including via apointing device, such as a mouse or trackpad, via menus, key commands,and so forth.

FIG. 13B illustrates graphical program 1302 after forces have beenapplied to the diagram or graphical program 1301 of FIG. 13A, where, asmay be seen, nodes 606 and 608 have been automatically moved to make wayfor node 610, wire 601 has been extended and moved accordingly, andnodes 602 and 604 repositioned towards the center of the modifieddiagram. As noted above, in some embodiments, the method may present ananimation or movie that illustrates transition from the diagram of FIG.13A to that of FIG. 13B.

As explained in more detail below, in some embodiments, the editing,adjusting, and displaying of the adjusted graphical diagram (e.g.,program) may be separate discrete events or processes, where, forexample, the user may move an element to a new position, after which theforces are computed and applied to other elements in the diagram (butinvisibly to the user), and then the resulting adjusted diagram isdisplayed, such that intermediate states of the diagram are notpresented to the user.

In contrast, in other embodiments, at least a portion of the editing,adjusting, and displaying the adjusted edited graphical program areperformed concurrently, e.g., in an interleaved manner, where, followingthe movement example above, as the user starts moving the element,corresponding forces are computed and applied to the other elements(based on the change of position so far) and the resulting incrementallyadjusted diagram displayed to the user. As the movement (or moregenerally, the editing operation) continues, the forces are continuallyupdated and applied, and successive states of the adjusted diagramdisplayed. Note, however, that in some embodiments, the editing may be asimple discrete event, e.g., if the user simply deletes an element inone action, and so the edit operation may be over by the time the forceschange as a result, but the application of resulting net forces,adjustments, and display of the adjusted diagram may unfold dynamically,e.g., smoothly, as the diagram or model approaches equilibrium, thusanimating the evolution of the diagram to equilibrium.

In some embodiments, one or more auxiliary features, components, orelements, invoked on, from, or associated with, a graphical program nodeor other element may be considered as an (possibly temporary) editoperation, and thus may instigate or invoke a corresponding applicationof forces. For example, user selection of a node may invoke display ofone or more selectable elements that may operate to configure,manipulate, access properties of, delete, or otherwise modify the node.In some embodiments, such elements may be referred to as “affordances”,although any other names may be used as desired. In one embodiment,selection of a node may invoke enlargement of the node to facilitatefurther user interaction with the node, e.g., to display one or moreaffordances. Such affordances and enlargements may result in automaticrearrangement of the graphical program via application of forces, asdescribed herein. Similarly, in some embodiments, when the node isde-selected, operations with the affordance or completed, or when someother suitable state is achieved, the affordance may be hidden orremoved, and the node may be compacted or contracted, in response towhich the graphical program may again be rearranged via application offorces.

FIGS. 14A-14C illustrate an exemplary embodiment of a graphical program1401 which is similar to the graphical program 600 of FIG. 6, andparticularly illustrates enlargement of a graphical program element,including display of an affordance and terminals, connection of theelement to another element, and subsequent rearrangement of programelements, including compaction of the graphical program element,according to one embodiment.

In this exemplary case, the absolute value node 610 has been selected,e.g., via a single tap, and in response, the node has been enlarged anda “delete” affordance 1410 displayed, where the delete affordance isselectable by the user to delete the node from the graphical program. Asalso shown, the selection of the node 610 has also invoked display ofrespective node terminals near the corners of the node, where those onthe left of the node are input terminals, and that on the right is anoutput terminal. Note that the node has been enlarged to make userinteraction with the node easier.

Accordingly, FIG. 14B illustrates user specification (again by gesture)of a wire between the output terminal of the node 610 and node 608,where the user has selected the output terminal and moved or dragged theterminal proximate to node 608 via a gesture, as indicated by the dashedcircle. Thus, the user has specified an edit operation where node 610 isconnected to node 608.

FIG. 14C illustrates the graphical program of FIGS. 14A and 14B afterforces have been applied in response to the edit operation of FIG. 14B,resulting in (edited) graphical program 1403. Note that once the editoperation was performed and implemented, the node 610 was reduced backto its normal size (compaction), and the various auxiliary elements,e.g., affordances, removed from the display, specifically, the deleteaffordance and terminals have been removed from the display. Moreover,application of the forces has resulted in node 610 being automaticallypositioned to make the wire joining it to node 608 straight, which issimpler and possibly more pleasing aesthetically.

Note that other affordances may operate differently. For example, anexemplary affordance that facilitates configuration of a node may bedisplayed in response to a single tap on a node (which may also invokeenlargement of the node). The affordance may be a symbol, icon, orlabel, displayed on or near the associated node, and may indicate themeaning or nature of the affordance. A single tap on the affordance mayinvoke display of a dialog, menu, palette, etc., via which the user mayconfigure the node. In some embodiments, the display of such elements,e.g., dialogs, menus, palettes, etc., may not perturb the graphicalprogram (although display of the affordance may, as noted above). Inother words, while introduction and display of some elements may perturbthe geometry of the program, and thus result in force-mediatedrearrangement of program elements, the addition and display of otherelements may not; rather, displays of these non-perturbing elements maybe overlaid on or near their associated nodes (or other elements)without introducing imbalances or perturbations in the model forces.

In some embodiments, one or more of the above-described effects orelements may be time limited. For example, when display of anaffordance, enlargement, etc., is invoked, a timer may be initiated, andupon expiration of a specified duration, the effect, e.g., affordance,enlargement, etc., may be removed, deactivated, etc. However, in someembodiments, user interaction with the node or associated effect (e.g.,affordance) may reset the timer.

A graphical user interface (GUI) may be provided whereby physical (i.e.,physics based) parameters and/or constraints related to diagram elementsand/or the diagram in general may be specified. Examples of configurableparameters include, but are not limited to, mass, viscosity, friction,maximum velocity, maximum (and/or minimum) repulsive or attractiveforce, model timestep or update frequency, and animationtimestep/framerate (draw rate), among others. In various embodiments, aGUI may be invoked at the element level (e.g., invoked via an affordanceassociated with the element, pop-up menu, etc.), at the diagram orprogram (e.g., global) level, or even at the development environmentlevel, e.g., where, for example, default values may be specified for anydiagrams or programs in development. Any types of GUI may be used asdesired, where the GUI may include one or more controls and/orindicators for manipulating and/or viewing parameter values. Thus, themethod may further include displaying a graphical user interface (GUI)that includes one or more controls or indicators. User input may bereceived specifying or modifying one or more of at least one parameterof the one or more physics based parameters, or at least one qualitativeaspect of the model or model behavior. Executing the model may then beperformed (or resumed or continued) in accordance with the specified ormodified at least one parameter or at least one qualitative aspect. Notethat in some embodiments, the model may be paused for such parametermodification, while in other embodiments, the displaying a GUI andreceiving user input specifying or modifying may be performeddynamically during execution of the model.

In one exemplary embodiment, a GUI may present qualitative controls forspecifying one or more parameters and/or the behavior of the model. Forexample, a GUI may present a “slider” control whereby a user may specifya value or level in a specified range for a user-perceivable attribute,e.g., a “slower/faster” control to determine how quickly elements maymove towards equilibrium. Thus, the user may control a qualitativeattribute without having to understand or even be aware of the one ormore quantitative values involved in producing the qualitativeattribute. Following the “slower/faster” example, in one embodiment,specifying a level of “speed” for the program model may in turnautomatically specify respective levels of multiple physics parameters,e.g., maximum velocity or acceleration, mass, etc. In some embodiments,a user may optionally invoke display of these implied parameter values,and so may more readily understand the underlying mechanisms and resultsof such qualitative controls. In this manner, a user may productivelymanage or specify the model's convergence to equilibrium withoutunderstanding the more detailed or subtle nuances of the process ofconvergence (to equilibrium), but may learn the nature of these nuanceswith use, and thus become more sophisticated over time, e.g., specifyingindividual parameter values in addition to or instead of suchqualitative attributes.

Further Embodiments

The following describes further exemplary embodiments of the techniquesdescribed herein. More specifically, further embodiments are describedregarding the nature and relationship of and between the application offorces in the model of the graphical diagram (e.g., graphical program),and animation of the transition from an initial state when the edit ismade to a final (equilibrium) state resulting from the application offorces invoked by the edit. It should be noted, however, that thevarious embodiments described are meant to be exemplary only, and arenot intended to limit the embodiments to any particular set of features.

In one embodiment, forces associated with the graphical diagram mayalways be active. For example, the method may run the simulation of thediagram as long as the graphical diagram is displayed in an editor. Notethat since movement of diagram elements may stop upon reachingequilibrium, i.e., upon reaching a state in which all forces are inbalance, a user may not be aware that a model of the graphical diagramis executing until a new edit operation occurs or is invoked. Thus, insome embodiments, the method may further include repeating thedetermining, summing, and moving in an iterative manner prior to thereceiving user input and after the equilibrium condition obtains, i.e.,in a substantially continuous manner. Accordingly, displaying thegraphical diagram and displaying the adjusted edited graphical programmay be subsumed by displaying an ongoing animation of the diagram, whereno movement is performed or illustrated unless some perturbation of thediagram geometry occurs, e.g., in response to an edit operation, orunless the diagram is not in an equilibrium condition or state, althoughthe model may continue to execute, sum forces, etc. In other words,displaying the animation may include the displaying of the graphicaldiagram (502), illustrating the moving (as forces are applied), anddisplaying the adjusted edited graphical diagram (508), e.g., ananimation may be displayed, e.g., in the edit window, illustrating themovement of the elements to an equilibrium state in which the forcesbalance and movement ceases.

As noted above, in some embodiments, at least a portion of the editing,adjusting, and displaying the adjusted edited graphical program may beperformed concurrently, or in an interleaved manner. For example, in anexemplary case where the user input moves a node from a first locationto a second location, as soon as the node's movement begins, resultingnet forces may be determined and the adjusting may also beginaccordingly, e.g., other nodes may start moving in response to themoving of the node, and may also be displayed accordingly, and so on inan iterative manner. When the node reaches the second location, theforce determination, adjustments, and display may be continued, therebyallowing the diagram elements to reach equilibrium. As noted elsewhere,in some embodiments, the force determination and displaying of theadjusted diagram may be performed continuously even after equilibrium isreached, but since the forces are balanced (due to equilibrium beingattained), no adjustments, and thus, no changes in the display, mayoccur, and thus it may not be apparent to the user that the physicssimulation is still executing.

In other embodiments, the editing, adjusting, and displaying may bediscrete events or operations. For example, the user may edit thegraphical diagram, e.g., moving a node to a new location, after whichthe forces may be calculated and applied “behind the scenes”, and onceequilibrium is reached, the adjusted diagram may be displayed. Thus,from the user's perspective, the displayed diagram may not evolvesmoothly, even though the underlying or associated model may do so,i.e., the displayed diagram may “jump” from the edited diagram image tothe final adjusted diagram image wherein all motion of diagram elementshas already ceased.

Alternatively, in some embodiments, the model may begin execution inresponse to an edit operation, e.g., a perturbation of the diagramlayout or geometry, and may cease or pause execution of the model uponreaching equilibrium, e.g., the modeling aspect may enter a “sleep”mode, and may be awakened by the next edit/perturbation. Note that“reaching equilibrium” may mean approaching a state that is within somespecified tolerance or “epsilon” of the ideal equilibrium state. Forexample, generally, as an element approaches its equilibrium state(position), the forces, and thus, the resulting velocity, may approachzero, e.g., asymptotically, and depending on the resolution of the data(e.g., 32 bit, 64 bit, etc.) may require significant time to actuallystop (come to rest), or may tend to “wiggle” indefinitely around theequilibrium value. Thus, in one embodiment, the method may ceaseapplying forces (and thus stop moving the elements) when some value ormetric achieves some specified threshold, e.g., when the largest currentmovement, velocity, or acceleration in the model is less than or equalto a specified threshold. Note that in some embodiments, ceasingapplication of forces may be achieved by setting all forces to zero.

As noted above, in some embodiments, the graphical program or diagramcan be stable in multiple different arrangements. Thus, in response toan edit operation (or other perturbation) that changes the geometry ofthe graphical diagram (including changes in individual elements), themethod may execute (or continue executing) the model to find the nearestequilibrium state (with respect to the current state). As discussed indetail above, examples of such edits or perturbations may include suchoperations as adding, removing, connecting, or moving, a graphicaldiagram element (e.g., icon, node, frame, structure, boundary, wire,edge, etc.). Further examples of such operations include, but are notlimited to, selecting, enlarging, compacting or contracting, expanding,collapsing, or resizing an element in the graphical diagram, displayingan affordance of an element in the graphical diagram, or displaying aterminal of an element in the graphical diagram, among others. Note thatexpanding and collapsing are actions or operations regarding replacementin a diagram of an icon with its constituent elements (expansion), andreplacement of multiple elements, e.g., a program or subprogram, or moregenerally, a diagram or sub-diagram, with a representative icon(collapse), respectively. This is in contrast to enlarging andcompacting/contracting, in which an element, e.g., a graphical programnode, is enlarged in place (in a diagram) to facilitate user interactionwith the element, and may include (temporarily) increasing the size ofthe element, and displaying additional aspects, such as terminals andaffordances (enlargement), and returning an enlarged element to itsoriginal size/appearance (compaction/contraction). Note that in variousembodiments, the one or more elements may include one or more of: atleast one graphical program node, at least one wire connected to agraphical program node, or at least one terminal in the graphicalprogram, e.g., a terminal of a program node or a GUI or GUI element,e.g., of a front panel. More generally, the one or more elements mayinclude one or more of: at least one graphical diagram node, at leastone wire connected to a graphical diagram node, or at least one terminalin the graphical diagram, e.g., a terminal of a diagram node or a GUI orGUI element, e.g., of a front panel.

As noted above, in some embodiments, the “frame rate” (i.e., cyclefrequency or update rate) of the model/simulation may be different fromthat of the animation or movie, e.g., the “draw rate”. More generally,in some embodiments, there may be a nonlinear relationship between thesimulation rate and the animation frame rate. Additionally, in someembodiments, one or more of the physics parameters employed in the modelmay be time or context dependent. For example, at least one physicsparameter may be dynamically modified during execution of the model toachieve some desired effect, e.g., to make the animation and/or feedbackfrom editing more pleasing or effective to a user. In other words,certain actions or movements may be more pleasing or understandable tousers, and one or more physical parameters may be modified dynamicallyto implement these actions or movements.

As one example, viscosity (e.g., resistance to motion), friction, orother damping parameter, directed to a given element (or multipleelements) may be dynamically modified, e.g., increased, as an elementapproaches an equilibrium position or zero velocity to more rapidlybring the element to rest at its final state and avoid oscillationsabout the equilibrium state. As a further example, friction (orviscosity) may be a function of the velocity, or the square of velocity,and so may operate to limit acceleration. In yet another example, a“temperature” of the graphical diagram or model may be defined thatindicates, and possibly bounds, the overall energy of motion of theconstituent elements, and may be manipulated to effect, e.g., lessen ornormalize, the time of convergence to equilibrium. Note that the aboveexamples are exemplary only, and that in various embodiments, any of thephysics parameters of the model may be dependent on, or manipulateddynamically based on, time, context, or any other parameter or aspect ofthe model or its elements, as desired.

In some embodiments, one or more physics based parameters of thegraphical diagram or graphical diagram elements, e.g., velocity, maximumvelocity, acceleration, maximum acceleration, etc., may have respectiveprofiles that specify values for the parameters over time, e.g., overthe duration of the convergence process, or with respect to some otherattribute or parameter. The method may thus include applying respectiveprofiles to the one or more physics based parameters of the graphicaldiagram or graphical diagram elements, wherein each profile specifies asequence of values for a respective parameter. In some embodiments, suchprofiles may be normalized, and may be scaled as needed for applicationto elements to effect a trajectory to equilibrium. Thus, for example,the same velocity profile may be scaled differently, e.g., time-wise,for application to different elements.

The method may “auto-tune” the model to improve the user's editingexperience. For example, the model may be tuned automatically to achievecritical damping or time-normalization of the convergence process.Critical damping refers to a configuration where a system reachesequilibrium most quickly without oscillations. Time-normalization refersto the imposition of a specified duration for a process, e.g., theconvergence to equilibrium process. For example, a user (or a defaultsetting) may specify the duration to be, e.g., 2 seconds, and so theconvergence process may be managed or manipulated in such a way thatconvergence to equilibrium occurs by approximately this time, e.g.,within some specified tolerance. There are a number of different waysthis can be accomplished.

In some embodiments, auto-tuning may include computing or estimating ametric or figure of merit that represents or indicates the degree towhich the graphical diagram is arranged in an optimal or desired manner,e.g., with respect to simplicity, clarity, compactness, and so forth.Such arrangements may be considered target states, and may be equivalentto or may correspond to equilibrium states of the model. Thus, themethod may determine or estimate a value of the figure of merit (ormetric) corresponding to a target state, and by comparing the currentvalue for the diagram or model, may determine a (possibly rough)estimate of the “distance” from equilibrium of the diagram or model. Themethod may then automatically tune or adjust various of the physicsparameters of the model to control or optimize convergence toequilibrium. In some embodiments, a state space may be defined for themodel, and the figure of merit, equilibrium points/target states, anddistances may be determined in or with respect to the state space.Moreover, in some embodiments, equilibrium values and distances to suchvalues may be determined empirically. For example, data regarding manydiagrams, edits, convergence histories, and/or equilibrium points, andpossibly user feedback re animations, may be collected from many users,e.g., at a server, and may be used to develop heuristics regarding thefigure of merit/metric, equilibrium points and distances thereto,convergence times, and parameter profiles, among other aspects of thetechniques disclosed herein. For example, the data may be (automaticallyor manually) analyzed to determine a set of “atomic” or fundamentalscenarios or diagram configurations and corresponding values (ofparameters, times, distances, etc.), which may be used to estimatevalues for combinations of these scenarios or configurations. In oneembodiment, a diagram may be analyzed and decomposed to its constituentfundamental scenarios or configurations, which may then be used todetermine such values of the diagram in general.

In some embodiments where computation resources are such that the modelexecutes faster than real time, in response to a perturbation (e.g., anedit operation), the method may determine or retrieve a value of thefigure of merit corresponding to an equilibrium or target state, or maydetermine the equilibrium or target state itself. For example, themethod may execute the model to converge to an equilibrium or targetstate, e.g., in the background, so that the model execution is invisibleto the user. The method may log or otherwise store some or all of theintermediate states of the model/diagram (e.g., positions, velocities,forces, sizes, and any other relevant attributes or parameters of thegraphical diagram, including those of the graphical diagram or diagramelements), and may then produce an animation that illustrates theconvergence to equilibrium, e.g., by sampling the sequence of states,and drawing the graphical diagram in accordance with the sampledsequence of states. More specifically, in some embodiments, e.g., whereexecuting the model of the graphical diagram is performed faster thanreal time, the method may include saving a sequence of model statesduring execution of the model, and displaying an animation of thegraphical diagram may include sampling the sequence of model states,thereby generating a sequence of sampled states, and displaying asequence of images of the graphical diagram corresponding to thesequence of sampled states.

In other words, the method may selectively animate the convergenceprocess in accordance with the sampled states. Note that the samplingand/or animation may not be linear with respect to the actual sequenceof model states. For example, in an exemplary case where an elementmoves a great distance, most of the movement may not be interesting to auser, and so the method may extract (e.g., sample) and animate orillustrate a sparse set of states covering this less interesting portionof the model evolution, but may extract and animate a denser set ofstates covering the more interesting final evolution to equilibrium.Thus, in this exemplary embodiment, the animation displayed to the usermay traverse the sparse set of states quickly, and then show the finalapproach to equilibrium in (relative) slow motion. More generally, themethod may sample the state sequence of the model, and may map (possiblynonlinearly) the state samples to an animation of the desired duration.Note that this technique manipulates or “warps” the model time in anonlinear fashion to achieve a pleasing or more engaging animationsequence. In some embodiments, the sequence of sample states (oranimation frames) may be interpolated to smooth the animation.

In an alternate embodiment, the sequence of model states may itself besaved in a nonlinear manner with respect to time. For example, based onone or more specified attributes, e.g., velocities, movements,temperature or entropy, or any other attributes or parameters, key modelstates reflecting or corresponding to interesting or important aspectsof the diagram's evolution to equilibrium may be saved and used togenerate the animation, where the method interpolates between keyanimation frames corresponding to the key model states. In a furtherembodiment, the saved sequence of model states (or a sampled subset) maybe mapped nonlinearly to animation frames, where more interesting orimportant portions of the model evolution are animated or illustrated athigher time resolutions (e.g., frames per second) than less interestingor important portions.

Note that a similar effect may be implemented via parameter profiles orother dynamic parameter adjustments, mentioned above. For example, toachieve a specified animation duration (e.g., 2 seconds), the method maydetermine, monitor, or track the distance from equilibrium, and maydynamically modify physical parameters during the convergence process toreach equilibrium at or near the specified deadline. Alternatively, oneor more parameter profiles may be determined or selected based on thedistance to equilibrium, desired duration of the animation, etc., andthe model configured (possibly dynamically) and executed accordingly. Insome embodiments, some combination of these techniques may be used.Thus, for example, the method may sample the state sequence of themodel, and may map the state samples to an animation of the desiredduration and in accordance with one or more acceleration or velocityprofiles (or profiles of other parameters or attributes as desired).

Note that the above techniques are not limited to graphical programediting, but are broadly applicable to any type of diagram that issubject to perturbations to its geometry, and for which clarity of (userperceptions of) the diagram is desired. Thus, for example, the abovetechniques may be applied to system diagrams, physical diagrams,configuration diagrams, etc., where system components, e.g., hardwareand/or software components, are linked together in particular ways, asnoted above.

FIG. 15 illustrates one embodiment of a configuration diagramrepresenting a distributed hardware system. As may be seen, thisparticular example system includes a plurality of devices coupled via avariety of communication buses, including Ethernet (Internet), GPIB,Wireless (e.g., Ethernet, Bluetooth), PCI, Serial I/O, USB, and 1394, asillustrative examples of inter-device connectivity means, although anyother devices, buses, and protocols may be used as desired. Note thatthis exemplary diagram also includes a software icon labeled “JellyBeans”. In some embodiments, software icons may be displayed connectedor proximate to hardware device icons to indicate that the correspondingsoftware is deployed to or stored on the corresponding hardware device.

Any of the techniques disclosed above with regard to graphical programsmay be applied to such diagrams and graphical diagrams in general, in asimilar manner. More generally, any of the techniques disclosed hereinmay be used in any combination desired, with respect to any graphicaldiagrams desired.

Although the embodiments above have been described in considerabledetail, numerous variations and modifications will become apparent tothose skilled in the art once the above disclosure is fully appreciated.It is intended that the following claims be interpreted to embrace allsuch variations and modifications.

I claim:
 1. A non-transitory computer-accessible memory medium thatstores program instructions executable by a processor to implement:displaying a graphical diagram on a display device, wherein thegraphical diagram comprises a plurality of interconnected nodes thatvisually indicate functionality of the graphical diagram; receiving userinput editing the graphical diagram, thereby generating an editedgraphical diagram; automatically adjusting placement of one or moreelements within the edited graphical diagram based on said editing,wherein said adjusting is performed based on determined forces appliedto the one or more elements in the edited graphical diagram, whereinsaid adjusting placement results in an adjusted edited graphicaldiagram, wherein said automatically adjusting comprises: for a firstelement of the one or more elements, determining one or more attractiveor repulsive forces based on the position of the first element and atleast one other element; and automatically adjusting the position of thefirst element based on the one or more attractive or repulsive forces;and displaying the adjusted edited graphical diagram on the displaydevice.
 2. The non-transitory computer accessible memory medium of claim1, wherein at least a portion of said editing, said adjusting, and saiddisplaying the adjusted edited graphical diagram are performedconcurrently.
 3. The non-transitory computer accessible memory medium ofclaim 1, wherein in implementing said adjusting placement, the programinstructions are further executable to implement: determining the forcesapplied to the one or more elements in the edited graphical diagrambased on the edit operation; calculating adjustments in placement of theone or more elements within the edited graphical diagram based onapplication of the determined forces to the one or more elements.
 4. Thenon-transitory computer accessible memory medium of claim 1, whereinsaid adjusting placement of one or more elements comprises: determiningone or more forces for each of the one or more elements based on theirpositions; summing the forces on each of the one or more elements,thereby determining resultant forces; moving the one or more elementsbased on the resultant forces; and repeating said determining, saidsumming, and said moving one or more times in an iterative manner untilan equilibrium condition obtains.
 5. The non-transitory computeraccessible memory medium of claim 4, wherein said displaying comprises:displaying an animation that illustrates said moving.
 6. Thenon-transitory computer accessible memory medium of claim 5, wherein theprogram instructions are further executable to perform: repeating saiddetermining, said summing, and said moving in an iterative manner priorto said receiving user input and after the equilibrium conditionobtains; wherein said displaying the animation further comprises:displaying the graphical diagram during said repeating said determining,said summing, and said moving in an iterative manner prior to saidreceiving user input and after the equilibrium condition obtains.
 7. Thenon-transitory computer accessible memory medium of claim 4, whereinsaid determining, said summing, and said moving one or more times in aniterative manner comprises: executing a model of the graphical diagram,wherein the model specifies and implements physics based attributes andbehaviors of the graphical diagram or elements of the graphical diagramvia a plurality of physics based parameters.
 8. The non-transitorycomputer accessible memory medium of claim 7, wherein said executing themodel comprises: dynamically modifying one or more of the physics basedparameters of the graphical diagram or elements of the graphicaldiagram.
 9. The non-transitory computer accessible memory medium ofclaim 8, wherein said dynamically adjusting one or more physics basedparameters comprises: applying respective profiles to the one or morephysics based parameters of the graphical diagram or elements of thegraphical diagram, wherein each profile specifies a sequence of valuesfor a respective parameter.
 10. The non-transitory computer accessiblememory medium of claim 8, wherein the program instructions are furtherexecutable to perform: displaying a graphical user interface (GUI)comprising one or more controls or indicators; and receiving user inputspecifying or modifying one or more of: at least one parameter of theone or more physics based parameters; or at least one qualitative aspectof the model or model behavior; wherein said executing the model isperformed in accordance with the specified or modified at least oneparameter or at least one qualitative aspect.
 11. The non-transitorycomputer accessible memory medium of claim 8, wherein said displaying aGUI and said receiving user input specifying or modifying is performeddynamically during said executing the model.
 12. The non-transitorycomputer accessible memory medium of claim 7, wherein the programinstructions are further executable to perform: repeating saiddetermining, said summing, and said moving in an iterative manner priorto said receiving user input and after the equilibrium conditionobtains; and displaying an animation of the graphical diagram,including: said displaying the graphical diagram; illustrating saidmoving; and said displaying the adjusted edited graphical diagram. 13.The non-transitory computer accessible memory medium of claim 12,wherein said executing the model of the graphical diagram is performedfaster than real time, wherein the program instructions are furtherexecutable to perform: saving a sequence of model states duringexecution of the model; wherein said displaying an animation of thegraphical diagram comprises: mapping the sequence of model states to asequence of images of the graphical diagram; and displaying the sequenceof images of the graphical diagram corresponding to the sequence ofmodel states.
 14. The non-transitory computer accessible memory mediumof claim 13, wherein said mapping comprises a nonlinear mapping.
 15. Thenon-transitory computer accessible memory medium of claim 12, whereinsaid executing the model of the graphical diagram is performed fasterthan real time, wherein the program instructions are further executableto perform: saving a sequence of model states during execution of themodel; wherein said displaying an animation of the graphical diagramcomprises: sampling the sequence of model states, thereby generating asequence of sampled states; and displaying a sequence of images of thegraphical diagram corresponding to the sequence of sampled states. 16.The non-transitory computer accessible memory medium of claim 1, whereinthe user input specifies or manipulates a graphical diagram element inthe graphical diagram.
 17. The non-transitory computer accessible memorymedium of claim 1, wherein the user input invokes or specifies one ormore of: selecting an element in the graphical diagram; adding anelement to the graphical diagram; removing an element from the graphicaldiagram; connecting two or more elements in the graphical diagram;moving an element in the graphical diagram; enlarging an element in thegraphical diagram; compacting an element in the graphical diagram;expanding an element in the graphical diagram; collapsing an element inthe graphical diagram; resizing an element in the graphical diagram;displaying an affordance of an element in the graphical diagram; ordisplaying a terminal of an element in the graphical diagram.
 18. Thenon-transitory computer accessible memory medium of claim 1, wherein theone or more elements comprise one or more of: at least one graphicaldiagram icon; at least one edge connected to a graphical diagram icon;or at least one terminal in the graphical diagram.
 19. Thenon-transitory computer accessible memory medium of claim 1, wherein thegraphical diagram comprises a graphical data flow block diagram.
 20. Thenon-transitory computer accessible memory medium of claim 1, wherein thegraphical diagram comprises a graphical control flow block diagram. 21.A method, comprising: utilizing a computer to perform: displaying agraphical diagram on a display device, wherein the graphical diagramcomprises a plurality of interconnected nodes that visually indicatefunctionality of the graphical diagram; receiving user input editing thegraphical diagram, thereby generating an edited graphical diagram;applying forces to one or more elements in the edited graphical diagrambased on said editing, wherein said applying forces operates toautomatically adjust placement of the one or more elements within theedited graphical diagram, resulting in an adjusted edited graphicaldiagram, wherein said applying forces comprises: for a first element ofthe one or more elements, determining one or more attractive orrepulsive forces based on the position of the first element and at leastone other element; and automatically adjusting the position of the firstelement based on the one or more attractive or repulsive forces; anddisplaying the adjusted edited graphical diagram on the display device.22. A system, comprising: a processor; and a memory, coupled to theprocessor, wherein the memory stores program instructions executable bythe processor to: display a graphical diagram on a display device,wherein the graphical diagram comprises a plurality of interconnectednodes that visually indicate functionality of the graphical diagram;receive user input editing the graphical diagram, thereby generating anedited graphical diagram; automatically adjust placement of one or moreelements within the edited graphical diagram based on determined forcesapplied to the one or more elements in the edited graphical diagrambased on said editing, resulting in an adjusted edited graphicaldiagram, wherein said automatically adjusting comprises: for a firstelement of the one or more elements, determining one or more attractiveor repulsive forces based on the position of the first element and atleast one other element; and automatically adjusting the position of thefirst element based on the one or more attractive or repulsive forces;and display the adjusted edited graphical diagram on the display device.